Thoracic Aortic Aneurysm and Dissection (TAAD) is one of the life-threatening cardiovascular diseases, and the histological changes of Medial Degeneration (MD) have important clinical significance for the diagnosis and early intervention of TAAD. Focusing on the issue that the diagnosis of MD is time-consuming and prone to poor consistency because of the great complexity in histological images, a deep learning based classification method of histological images was proposed, and it was applied to four types of MD pathological changes to verify its performance. In the method, an improved Convolutional Neural Network (CNN) model was employed based on the GoogLeNet. Firstly, transfer learning was adopted for applying the prior knowledge to the expression of TAAD histopathological images. Then, Focal loss and L2 regularization were utilized to solve the data imbalance problem, so as to optimize the model performance. Experimental results show that the proposed model is able to achieve the average accuracy of four-class classification of 98.78%, showing a good generalizability. It can be seen that the proposed method can effectively improve the diagnostic efficiency of pathologists.